Wide-band multi-format audio/video production system with frame-rate conversion

ABSTRACT

A multi-format digital video production system enables a user to process an input video program to produce an output version of the program in a final format which may have a different frame rate, pixel dimensions, or both. An internal production format of 24 fps is preferably chosen to provide the greatest compatibility with existing and planned formats associated with HDTV standard 4:3 or widescreen 16:9 high-definition television, and film. Images are re-sized horizontally and vertically by pixel interpolation, thereby producing larger or smaller image dimensions so as to fill the particular needs of individual applications. Frame rates are adapted by inter-frame interpolation or by traditional schemes, including “3:2 pull-down” for 24-to-30 fps conversions. Simple speed-up (for 24-to-25 conversions) or slow-down (for 25-to-24 conversions) for playback, or by manipulating the frame rate itself using a program storage facility with asynchronous reading and writing capabilities. The step of converting the signal to a HDTV format is preferably performed using a modified upconversion process for wideband signals (utilizing a higher sampling clock frequency) and a resizing to HDTV format frame dimensions in pixels.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/348,804 filed Jan. 5, 2009 which is a continuation of U.S. patentapplication Ser. No. 10/117,496 filed Apr. 5, 2002; which is acontinuation of U.S. patent application Ser. No. 09/305,953 filed May 6,1999, now U.S. Pat. No. 6,370,198; which is a continuation-in-part ofU.S. patent application Ser. No. 08/834,912, filed Apr. 7, 1997, nowU.S. Pat. No. 5,999,220; and also claims priority of U.S. ProvisionalPatent Application Ser. No. 60/084,522, filed May 7, 1998. All of theseprior applications are incorporated herein by reference in theirentirety.

FIELD OF THE INVENTION

This invention relates generally to video production, photographic imageprocessing, and computer graphics, and, more particularly, to amulti-format digital video production system capable of maintaining thefull bandwidth resolution of the subject material, while providingprofessional quality editing and manipulation of images intended fordigital television and other applications, including digital HDTVprograms.

BACKGROUND OF THE INVENTION

As the number of television channels available through various programdelivery methods digital TV (DTV) broadcasting, cable TV, home video,broadcast, etc. continues to proliferate, the demand for programming,particularly high-quality HDTV-format programming, presents specialchallenges, both technical and financial, to program producers. Whilethe price of professional editing and image manipulation equipmentcontinues to increase, due to the high cost of research and developmentand other factors, general-purpose hardware, including personalcomputers, can produce remarkable effects at a cost well within thereach of non-professionals, even novices. As a result, the distinctionbetween these two classifications of equipment has become less welldefined. Although general-purpose PC-based equipment may never allowprofessional-style rendering of images at full resolution in real-time,each new generation of microprocessors enables progressively faster,higher-resolution applications. In addition, as the price of memorycircuits and other data storage hardware continues to fall, the capacityof such devices has risen dramatically, thereby improving the prospectsfor enhancing PC-based image manipulation systems for such applications.

In terms of dedicated equipment, attention has traditionally focused onthe development of two kinds of professional image-manipulation systems:those intended for the highest quality levels to support film effects,and those intended for television broadcast to provide “full 35 mmtheatrical film quality,” within the realities and economics of presentbroadcasting systems. Conventional thinking holds that 35 mm theatricalfilm quality as projected in theaters is equivalent to 1200 or morelines of resolution, whereas camera negatives provide 2500 or morelines. As a result, image formats under consideration have been directedtowards video systems having 2500 or more scan lines for high-levelproduction, with hierarchies of production, HDTV broadcast, and NTSC andPAL compatible standards which are derived by down-converting theseformats. Most proposals employ progressive scanning, although interlaceis considered an acceptable alternative as part of an evolutionaryprocess. Another important issue is adaptability tocomputer-graphics-compatible formats.

Current technology directions in computers and image processing shouldallow production equipment based upon fewer than 12200 scan lines, withpicture expansions to create a hierarchy of upward-converted formats fortheatrical projection, film effects, and film recording. In addition,general-purpose hardware enhancements should be capable of addressingthe economic aspects of production, a subject not considered in detailby any of the available references.

For the first fifty years of television in the United States, thehistory shows continuous development and improvement of a purelyanalog-based system for video production broadcasting. The nature of theNTSC system is to limit the video bandwidth to 4.2 MHZ, whichcorresponds to approximately 340 TV-lines of resolution. In countrieswhere PAL or SECAM systems are employed, the bandwidth is 5.5 MHZ, whichcorresponds to approximately 440 TV-lines of resolution.

During the past ten years, digital processing has become the standardfor video production equipment. However, to preserve compatibility withexisting equipment and standards, the video bandwidth typically has beenlimited to 4-6 MHZ (for NTSC and PAL applications, respectively). Thisalso has tended to reduce the apparent generation loss during videoproduction steps.

In the past five years or so, digital image compression technology hasmatured greatly. Furthermore, there are many incompatible standards,such as the different forms of JPEG systems, the-Quick-Time system,MPEG-1, and the numerous forms of the MPEG-2 standard. In addition, thelatest recording formats for video production have introduced a new setof variations, including the ¼-inch DVC-formats from Sony andMatsushita. While the signal deterioration characteristics ofmulti-generation analog-based production systems are well known, thoseimperfections resulting from diverse-format digital video compressionand the conversions between these formats can be just as troublesome andunpredictable. In practice, these repeated steps of analog-to-digital(A/D) conversion and digital-to-analog (D/A) conversion, as well as datacompression and decompression, introduce many signal artifacts andvarious forms of signal noise. Although digital video productionpromises multiple-step production processes free of generation losses,the reality is different, due to the repeated steps of A/D and D/Aconversions, as well as data compression and decompression, present whenutilizing the various incompatible image data compression formats.

Meanwhile, during the last twenty years, camera technology has advancedto a point far surpassing the performance of traditional productionequipment. The video bandwidth capability has increased from 4.2 MHZ(corresponding to 340 TV-lines of resolution) to approximately 12 MHZ(corresponding to nearly 1000 TV-lines of resolution). Because of thelimitations of conventional broadcast and production equipment, most ofthe detail information produced by today's high-performance camerasystems is lost.

For HDTV systems, one goal is to produce images having approximately1000 TV-lines of resolution per picture height, which requires abandwidth of approximately 30 MHZ. This, in turn, raises a new problemin terms of signal-to-noise ratio. While conventional broadcast camerascan produce signals having a S/N ratio of 65 dB, utilizing 10-bitdigital processing, HDTV cameras typically produce signals having a S/Nratio of 54 dB, and utilize only 8-bit digital processing. In addition,the typical HDTV camera utilizes a 2 Megapixel CCD, in which theelements are approximately one-quarter the size of conventionalbroadcast cameras. This translates into a much lower sensitivity (a losscorresponding to 1-2 lens f-stops), higher levels of “smearing”, andlower highlight compression ratios.

Analog-based HDTV systems, such as the Japanese MUSE system, do notapproach the design goal of 1000 TV-lines. In reality, only one quarterof the picture information is transmitted. Although the nominal reducedluminance bandwidth of 20 MHZ provides approximately 600 TV-lines ofresolution per picture height in static program material, thisresolution is drastically reduced to only 450 TV-lines where motion isoccurring. The chrominance bandwidth is even further reduced by thesub-sampling scheme, to 280 TV-lines for the I-signal and 190 TV-linesfor the Q-signal (in static scenes), and to 140 TV-lines for theI-signal and 50 TV-lines for the Q-signal (in moving scenes). Althoughthis system provides a wide-screen aspect ratio of 16:9, it does notreally qualify as a High-Definition Television System.

Because of the aforementioned compatibility issues, it is clear thatconventional video recorders cannot match the technical performance ofmodern camera systems. Although “D-6 format” digital recorders areavailable, the cost and complexity of such equipment place these unitsbeyond the means of the vast majority of broadcast stations.Furthermore, the capability of conventional switchers and otherproduction equipment still fail to match that of available camerasystems.

Other recorders have been produced, such as the one-half-inch portablerecorder (“Uni-HI”), but this system only achieves 42 dB signal-to-noiseratio, and records in the analog domain. These specifications renderthis unit unsuitable for multi-generation editing applications.Furthermore, the luminance bandwidth is only 20 MHZ, corresponding toapproximately 600 TV-lines of resolution.

W-VHS (“Wideband-VHS”) recorders provide a wide aspect-ratio image, butonly 300 TV-lines of resolution, which also renders this unit unsuitablefor any professional applications. Other distribution formats (such asD-VHS) require the application of high compression ratios to limit thedata-rate to be recorded, so these formats only achieve W-VHS quality(less than 400 TV-lines of resolution).

The newly-introduced HD Digital Betacam format (HDCAM) video recorderutilizes a 3:1:1 digital processing system rather than the 4:2:2processing. However, it has a 24 MHZ luminance bandwidth correspondingto 700 TV-lines of resolution, and a narrower chrominance bandwidth.Although this system is clearly superior to any existing analog HDTVrecording system, it still falls short of delivering the full resolutionproduced by an HDTV digital camera. Because of its proprietary imagedata compression format, the production process results in repeated datacompression and decompression steps, as well as A/D and D/A conversions,which, in turn, results in many signal artifacts and various forms ofsignal noise.

In summary, the conventional technology for these markets utilizesprofessional cameras having a 30 MHZ bandwidth, and capable of 1000TV-lines of resolution. However, they produce quality levels morecharacteristic of consumer-grade equipment (in terms of resolution andsignal-to-noise ratio). In addition, the price of these systems iscost-prohibitive both on an absolute and also a cost/benefit basis,employing digital systems which produce only analog-type performance.

SUMMARY OF THE INVENTION

The present invention takes advantage of available general-purposetechnology, where possible, in order to provide an economicalmulti-format digital video production system. In the preferredembodiment, specialized graphics processing capabilities are included ina high-performance personal computer or workstation, enabling the userto edit and manipulate an input video program and produce an outputversion of the program in a final format which may have a differentframe rate, pixel dimensions, or both. An internal production format ischosen which provides the greatest compatibility with existing andplanned formats associated with HDTV standard 4:3 or widescreen 16:9high-definition television, and film. For compatibility with film, theframe rate of the internal production format preferably is 24 fps (forprogram materials originated in film format) and 48 fields-per-second(for live program materials such as sporting events). Images arere-sized horizontally and vertically by pixel interpolation, therebyproducing larger or smaller image dimensions so as to fill theparticular needs of individual applications. Frame rates are adapted byinter-frame interpolation or by traditional schemes, including “3:2pull-down” for 24-to-30 fps conversions. Simple speed-up (for 24-to-25conversions) or slow-down (for 25-to-24 conversions) for playback, or bymanipulating the frame rate itself using a program storage facility withasynchronous reading and writing capabilities. The step of convertingthe signal to a HDTV format is performed by a modified upconversionprocess for wideband signals (utilizing a higher sampling clockfrequency) and a resizing to HDTV format frame dimensions in pixels.

The invention preferably incorporates one or more interface units,including a standard/widescreen interface unit operative to convert thevideo program in the input format into an output signal representativeof a standard/widescreen formatted image, and output the signal to anattached display device. A high-definition television interface unit isoperative to convert the video program in the input format into anoutput signal representative of an HDTV-formatted image, and output thesignal to the display device. A centralized controller in operativecommunication with the video program input, the graphics processor, andan operator interface, enables commands entered by an operator to causethe graphics processor to perform one or more of the conversions usingthe television interfaces. The present invention thus encouragesproduction at relatively low pixel dimensions to make use of lower-costgeneral-purpose technology and to maintain high signal-to-noise ratio,and then subsequently expands the resultant image into a so-calledup-converted program. This is in contrast to alternative approaches,which recommend operating at HDTV-type resolution, then down-converting,as necessary, to smaller image formats. This has led to the use ofexpensive dedicated hardware, the need for which the present inventionseeks to eliminate. In addition, the flexible storage and playbackfacilities allow extensive control of the playback of the programmaterial, enabling frame rate adjustments and alterations, and providingfor time-shifting of the start and end points of the programreproduction in those cases wherein direct control of the sourcematerial frame rate is not practical, due to physical separation of theequipment or multiple reception points simultaneously producing outputsat different frame rates from the same source signal playback datastream. In commercial implementations, the invention readily accepts andprocesses enhanced information, such as pan/scan information oridentification information to restrict viewing based on regional orgeographical marketing plans.

The method and associated technology provide for maintaining theoriginal high bandwidth of conventional cameras (up to 15 MHZ, whichcorresponds to more than 600 TV-lines of resolution-per picture heightfor 16:9 aspect ratio) and provide optimized compression techniques tofully utilize the available capacity of general storage media, such asthe commercially available Panasonic DVCPRO, DVCPRO50, Sony DVCAM, JVCDigital-S, and Sony Betacam SX recorders. The system preferably employsa consistent compression scheme, utilizing only intra-frame compression(such as Motion-JPEG-type systems, systems used in DV-format recorders,MPEG-2 4:2:2 P@ML) throughout the entire production process. This avoidsmany signal artifacts, ensures high signal-to-noise ratios, and providesfor editing the program material in data-compressed format. This enablesthe system to preserve the original camera capability of 600+ TV-linesof resolution per picture height, and with 4:2:2 processing provides achrominance bandwidth of up to 7.5 MHZ. Utilizing 10-bit processingresults in 65 dB signal-to-noise performance and improved camerasensitivity (rating of f-11). In contrast, available and proposedsystems for HDTV are based on 8-bit processing, and offer performance ofless than 54 dB signal-to-noise ratio and camera sensitivity rating ofonly f-8.

The invention provides for optimization of the available storage mediaas well. Utilizing hard-disks, optical discs (such as DVD, DVD-R, andDVD-RAM), magneto-optical discs, or digital tapes (such as DAT-format,DVC, DVCPRO, DVCPRO50, DVCAM, Digital-S, or 8-mm format) the data-rateto be recorded is nearly one-quarter-that of conventional HDTV systems,and consumes only 20 GB of storage space to record more than 60 minutesin the Production Format compression scheme, which utilizes a data-rateof 50 Mb per second or less, which is well within the capabilities ofcertain conventional recording devices. Horizontal and verticalpixel-interpolation techniques are utilized to quadruple the image size,preferably resulting in an image frame size of 1920×1080 pixels. Theresulting program information may then be distributed in a conventionalcompression format, such as MPEG-2.

Three alternative image frame sizes preferably are suggested, dependingon the intended application. For general usage, an image frame size of1024×576 is recommended. As an option, a frame size of either 1280×720or 1920×1080 may be utilized, at 24 frames-per-second. A samplingfrequency of up to 74.25 MHZ for luminance is utilized for 1920×1080.Sampling frequencies of up to 37 MHZ are preferably are utilized for1024×576 and 128×720. Chrominance components preferably are sampledconsistent with a 4:2:2 system, and 10-bit precision is preferred.

The technology of display devices and methodology has progressed aswell, offering alternative features such as conversion of interlacedsignals to progressive scan, line doubling, pixel quadrupling, andimproved general techniques for horizontal and vertical pixelinterpolation. Availability of these features as part of display deviceswill simplify the process of implementing multi-format digitalproduction.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1D show the preferred and alternative image aspect ratios inpixels;

FIG. 2 shows a functional diagram for disk/tape-based video recording;

FIG. 3 shows the components comprising the multi-format audio/videoproduction system;

FIG. 4 is a block diagram of an alternative embodiment of video programstorage means incorporating asynchronous reading and writingcapabilities to carry out frame-rate conversions;

FIG. 5 shows the inter-relationship of the multi-format audio/videoproduction system to many of the various existing and planned videoformats;

FIG. 6 shows the implementation of a complete television productionsystem, including signals provided by broadcast sources, satellitereceivers, and data-network interfaces;

FIGS. 7A-7B show the preferred methods for conversion between several ofthe most common frame-rate choices;

FIGS. 7C-7I show details of possible methods for frame rate conversionprocesses; and

FIG. 8 shows a block diagram of an embodiment of a universal playbackdevice for multi-format use.

DETAILED DESCRIPTION OF THE INVENTION

The present invention resides in the conversion of disparate graphics ortelevision formats, including requisite frame-rate conversions, toestablish an inter-related family of aspect ratios, resolutions, andframe rates, while remaining compatible with available and futuregraphics/TV formats, including images of pixel dimensions capable ofbeing displayed on currently available multi-scan computer monitors.Custom hardware is also disclosed whereby frames of higher pixel-countbeyond the capabilities of these monitors may be viewed. Images arere-sized by the system to larger or smaller dimensions so as to fill theparticular needs of individual applications, and frame rates are adaptedby inter-frame interpolation or by traditional schemes such as using“3:2 pull-down” (such as 24 frame-per-second (fps) Progressive to 30 fpsinterlace shown in FIG. 7C or 48 fps Progressive to 60 fps Progressive,as would be utilized for film-to-NTSC conversions) or by speeding up theframe rate itself (such as for 24 to 25 fps for PAL television display).The re-sizing operations may involve preservation of the image aspectratio, or may change the aspect ratio by “cropping” certain areas, byperforming non-linear transformations, such as “squeezing” the picture,or by changing the vision center for “panning,” “scanning” and so forth.Inasmuch as film is often referred to as “the universal format,”(primarily because 35-mm film equipment is standardized and usedthroughout the world), the preferred internal or “production” frame rateis preferably 24 fps. This selection also has an additional benefit, inflint the 24 fps rate allows the implementation of cameras havinggreater sensitivity than at 30 fps, which is even more critical insystems using progressive scanning (for which the rate will be 48 fieldsper second interlaced (or 24 fps Progressive) vs. 60 fields per secondinterlaced in some other proposed systems).

The image dimensions chosen allow the use of conventional CCD-typecameras, but the use of digital processing directly through the entiresignal chain is preferred, and this is implemented by replacing thetypical analog RGB processing circuitry with fully digital circuitry.Production effects may be conducted in whatever image size isappropriate, and then re-sized for recording. Images are recorded bywriting the digital data to storage devices employing internal orremovable hard-disk drives, disk drives with removable media, optical ormagneto-optical based drives, DVD-R or DVD-RAM type drives, tape-baseddrives, or semiconductor-based memory devices, preferably incompressed-data form.

As data rates for image processing and reading from, or writing to, diskdrives increase, many processes that currently require several secondswill soon become attainable in real-time. This will eliminate the needto record film or video frames at slower rates. Other productioneffects, such as slow-motion or fast-motion may be incorporated, and itis only the frame-processing-rate of these effects that is limited inany way by the technology of the day. In particular, techniques such asnon-linear-editing, animation, and special-effects will benefit from theimplementation of this system. In terms of audio, the data raterequirements are largely a function of sound quality. The audio signalsmay be handled separately, as in an “interlocked” or synchronized systemfor production, or the audio data may be interleaved within the videodata stream. The method selected will depend on the type of productionmanipulations desired, and by the limitations of the current technology.

Although a wide variety of video formats and apparatus configurationsare applicable to the present invention, the system will be described interms of the alternatives most compatible with currently availableequipment and methods. FIG. 1A illustrates one example of a compatiblesystem of image sizes and pixel dimensions. The selected frame rate ispreferably 24 per second progressive (for compatibility with filmelements), or 48 fields per second interlaced (for live program materialsuch as sporting events). The selected picture dimension in pixels ispreferably 1024×576 (0.5625 Mpxl), for compatibility with the StandardDefinition TV (SDTV) 16:9 “wide-screen” aspect ratio anticipated forHDTV systems, and the conventional 4:3 aspect ratio used for PAL systems[768×576 (0.421875 Mpxl)] or NTSC systems [640×480 (0.3072 Mpxl)]. Allimplementations preferably rely on square pixels, though other pixelshapes may be used. Re-sizing (using the well known, sophisticatedsampling techniques available in many image-manipulation softwarepackages or, alternatively, using horizontal and vertical pixelinterpolation hardware circuitry described herein below) either to1280×720 (0.922 Mpxl) or else to 1920×1080 (2.14 Mpxl) provides an imagesuitable for HDTV displays or even theatrical projection systems, and afurther re-sizing to 3840×2160 (8.3 Mpxl) is appropriate for even themost demanding production effects. Images may be data compressed,preferably 5:1 with Motion-WEG-type compression such as utilized inDV-format equipment, or preferably 10:1 with MPEG2 4:2:2 P@MLcompression.

In order to preserve the full bandwidth of this high-resolution signal,a higher sampling frequency is required for encoding, preferablyapproximately 20 MHZ, for 1024×576 at 24 fps, which results in 1250samples per total line, with 625 total lines per frame. This samplingrate allows processing a 10 MHZ bandwidth luminance signal, whichcorresponds to approximately 600 TV lines of resolution per pictureheight. In contrast, traditional SDTV digital component systems employ asampling frequency of 13.5 MHZ, which provides a luminance bandwidth of5 to 6 MHZ (approximately 300 to 360 TV lines of resolution per pictureheight. These wide-band data files may then be stored on conventionalmagnetic or optical disk drives, or tape-based storage units, requiringonly approximately 5.5 MB/sec for SDTV wide-screen frames in Y/R-Y/B-Y(assuming a 4:2:2 system at 8 bits per sample). The resultant data ratefor this system is less than 50 Megabits per second, which is within thecapabilities of currently available video recording equipment, such asthe Betacam SX, DVCPRO50 or Digital S50. If a higher data-compressionratio is applied, then other units may be used, such as DVC, DVCPRO orDVCAM; Betacam SX, DVCPRO50 or Digital S50 may be used to allow samplingto 10-bit precision rather than 8-bit precision.

An alternative aspect of the invention is shown in FIG. 1B. In thiscase, the user follows a technique commonly used in film production, inwhich the film is exposed as a 4:3 aspect ratio image. When projected asa wide-screen format image, the upper and lower areas of the frame maybe blocked by an aperture plate, so that the image shows the desiredaspect ratio (typically 1.85:1 or 1.66:1). If the original image formatwere recorded at 24 frames per second, with a 4:3 ratio and with adimension in pixels of 1024×768, all image manipulations would preservethese dimensions. Complete compatibility with the existing formats wouldresult, with NTSC and PAL images produced directly from these images byre-scaling, and the aforementioned wide-screen images would be providedby excluding 96 rows of pixels from the top of the image and 96 rows ofpixels from the bottom of the image, resulting in the 1024×576 imagesize as disclosed above. The data content of each of these frames wouldbe 0.75 Mpxls, and the data storage requirements disclosed above wouldbe affected accordingly.

Another aspect of the invention is depicted in FIG. 1C. In thisalternative, the system would follow the image dimensions suggested inseveral proposed digital HDTV formats considered by the AdvancedTelevision Study Committee of the Federal Communications Commission. Theformat adopted assumes a wide-screen image having dimensions of 1280×720pixels. Using these image dimensions (but at 24 fps progressive),compatibility with the existing formats would be available, with NTSCand PAL images derived from this frame size by excluding 160 columns ofpixels from each side of the image, thereby resulting in an image havinga dimension in pixels of 960×720. This new image would then be re-scaledto produce images having pixel dimensions of 640×480 for NTSC, or768×576 for PAL. The corresponding wide-screen formats would be 854×480and 1024×576, respectively. Utilizing a 4:2:2 sampling scheme, the1280×720 image will require 1.85 MB when sampled at a precision of8-bits, and 2.3 MB when sampled at a precision of 10-bits. When thesesignals are data-compressed utilizing a compression ratio of 10:1 forrecording, the two image sizes require data rates of 4.44 MB per second(35.5 megabits per second) or 5.55 MB per second (44.4 megabits persecond).

In order to preserve the full 15 MHZ bandwidth of this high-resolutionsignal, a sampling frequency of approximately 30 MHZ is required forencoding, which results in 1650 samples per total line, with 750 totallines per frame for a 1280×720 image at 24 frames-per-second. Incontrast, typical high definition systems require sampling rates of 74MHZ to provide a bandwidth of 30 MHZ). In this case, an image having adimension in pixels of 1280×720 would contain 0.87890625 Mpxl, with 720TV lines of resolution. Furthermore, the systems under evaluation by theATSC of the FCC all assume a decimation of the two chrominance signals,with detail of only 640×360 pixels retained. Overall, the data rate forthis system, utilizing 4:2:2 sampling with 10-bit precision, is lessthan 50 megabits per second. This is within the capabilities ofcurrently available video recording equipment, such as Betacam SX, theDVCPRO50 or Digital S50. Because expensive, high data-rate recorders(such as the Toshiba D-6 format, the HDCAM, and D-5 format), are notrequired for applications utilizing the instant invention, the cost ofthe equipment and production systems for these applications isdrastically reduced. The development path to 24 fps progressive is bothwell-defined and practical, as is the use of the previously describedmethods to produce images having a dimension in pixels of 1920×1080.

A third embodiment of the invention is depicted in FIG. 1D. In thisalternative, the system would follow the image dimensions suggested inseveral proposed digital HDTV formats considered by the AdvancedTelevision Study Committee of the Federal Communications Commission. Theformat adopted assumes a wide-screen image having dimensions of1920×1080 pixels (2.1 megapixels), but at 24 frames-per-secondProgressive. Utilizing a 4:2:2 sampling scheme, this 1920×1080 imagewill require 4.2 MB when sampled at a precision of 8-bits, and 5.2 MBwhen sampled at a precision of 10-bits. When these signals aredata-compressed utilizing a compression ratio of 10:1 for recording, thetwo image sizes require data rates of 10 MB per second (80 Megabits persecond) or 12.5 MB per second (96 megabits per second). In order topreserve the full bandwidth of this high-resolution signal, a samplingfrequency of 74.25 MHZ is required for encoding, which results in 2750samples per total line, with 1125 total lines per frame. In this case,an image having these dimensions would have over 1,200 TV lines ofresolution per picture height, representing over 30 MHZ luminancebandwidth. The chrominance bandwidth (as R-Y/B-Y) would be 15 MHZ. Incontrast, HDTV with 1920×1080 and 30 fps Interlace only produces 1,000TV lines (200 lines less than above) of resolution per picture heightfrom same sampling frequency of 74.25 MHZ.

Overall, the data rate for this system, utilizing 4:2:2 sampling with10-bit precision, is less than 100 Megabits per second. This is withinthe capabilities of video recording equipment, such as the PanasonicDVCPRO100 or JVC Digital 5100, which will be available in the nearfuture. Because expensive, high data-rate recorders (such as the ToshibaD-6 format, the HDCAM, and D-5 format), are not required forapplications utilizing the instant invention, the cost of the equipmentand production systems for these applications is drastically reduced.These images may be resized into frames as large as 7680×4320, whichwould allow use of the system for special optical effects, or withother, specialized film formats, such as IMAX and those employing 65 mm.Camera negatives. In addition, conversions processes are available, asdescribed herein below, to produce other HDTV formats (such as 1280×720Progressive at 24 fps, 1920×1080 Interlaced at 25 fps, 1920×1080Progressive at 50 fps, 1920×1080 Interlaced at 30 fps, and 1920×1080Progressive at 60 fps), or to alternative SDTV formats, (such as1024×576 at 25 fps, 768×576 at 25 fps, 853×480 at 30 fps, or 640×480 at30 fps).

In each of the cases described herein above, a positioning or imagecentering signal may be included within the data stream, so as to allowthe inclusion of information which may be utilized by the receiving unitor display monitor to perform a “pan/scan” operation, and thereby tooptimize the display of a signal having a different aspect ratio thanthat of the display unit. For example, a program transmitted in awide-screen format would include information indicating the changingposition of the image center, so that a conventional (4:3 aspect ratio)display unit would automatically pan (horizontally and/or vertically) tothe proper location. For the display of the credits or special panoramicviews, the monitor optionally could be switched to a full “letter-box”display, or the image could be centered and resealed to includeinformation corresponding to an intermediate situation, such as halfwaybetween full-height (with cropped sides) and letter-box (full-width, butwith blank spaces above and below the image on the display). Thispositioning/resealing information would be determined under operatorcontrol (as is typical for pan/scan operations when performing filmtransfers to video) so as to maintain the artistic values of theoriginal material, within the limitations of the intended displayformat.

Conventional CCD-element cameras produce images of over 900 TV Lineshorizontal Luminance (Y) resolution, with a sensitivity of 2,000 lux atf-11, and with a signal-to-noise ratio of 65 dB. However, typical HDTVcameras, at 1,000 TV Lines resolution and with sensitivity ratings off-8, produce an image with only a 54 dB signal-to-noise ratio, due tothe constraints of the wideband analog amplifiers and the smallerphysical size of the CCD-pixel-elements. By employing the moreconventional CCD-elements in the camera systems of this invention, andby relying upon the computer to create the HDTV-type image by imagere-sizing, the improved signal-to-noise ratio is retained. In thepractical implementation of cameras conforming to this new designapproach, there will be less of a need for extensive lightingprovisions, which in turn, means less demand upon the power generatorsin remote productions, and for AC-power in studio applications.

In CCD-based cameras, it is also a common technique to increase theapparent resolution by mounting the red and blue CCD-elements inregistration, but offsetting the green CCD-element by one-half pixelwidth horizontally and in some application vertically. In this case,picture information is in-phase, but spurious information due toaliasing is out-of-phase. When the three color signals are mixed, thepicture information is intact, but most of the alias information will becanceled out. This technique will evidently be less effective whenobjects are of solid colors, so it is still the usual practice toinclude low-pass optical filters mounted on each CCD-element to suppressthe alias information. In addition, this technique cannot be applied tocomputer-based graphics, in which the pixel images for each color arealways in registration. However, for Y/R-Y/B-Y video, the result of theapplication of this spatial-shift offset is to raise the apparentLuminance (Y) horizontal resolution to approximately 900 televisionlines (a 4:3 aspect ratio utilizing 12200 active pixels per line), andthe apparent vertical resolution is increased by 50-100+ lines.

During the transition period to implement 24 fps recording as a newproduction standard, conventional 16:9 widescreen-capable CCD cameras(running in 25 or 30 fps Interlaced mode) may be utilized to implementthe wideband recording method so as to preserve the inherent widebandcapability of these cameras, in accordance with the invention. Byabandoning the requirement for square pixels, sampling frequencies of upto 30 MHZ for luminance (15 MHZ for chrominance) preferably areutilized, which frequencies are less than half the typical sampling rateof 74 MHZ utilized for typical HDTV luminance signals in alternativesystems. Chrominance components preferably are sampled consistent with a4:2:2 system. This wideband data stream is then compressed 10:1,utilizing MPEG2 4:2:2 P@ML at 10-bit. The resultant data rate is stillless than 50 Megabits per second. With a straightforward modification toincrease the data compression rate to 10:1, this signal may be recordedutilizing any of several conventional recording devices, includingPanasonic DVCPRO50, NC Digital-S, and Sony Betacam SX, therebypreserving the wideband signal (up to 800 TV lines of resolution perpicture height). By utilizing the appropriate techniques for imageresizing and frame rate conversion as described herein, video systemsmay be supported consistent with 1280×720 60 fps progressive, 1280×72024 fps Progressive, 1920×1080 25 fps Interlace, 1920×1080 30 fpsInterlace, 1920×1080 50 fps progressive, 1920×1080 60 fps progressive,in accordance with the invention.

The availability of hard-disk drives of progressively higher capacityand data transmission rates is allowing successively longer programduration and higher resolution image displays in real-time. At thepreviously cited data rates, wide-screen frames (1024×576 pixel, 24 fps,4:2:2 process, 8 bits precision and 5:1 compression) would require 330MB/min, so that currently available 10 GB disk drives will store morethan 30 minutes of video. When the anticipated 50 GB disk drives(5.25-inch disks) become available from Seagate within the year, theseunits will store 150 minutes, or 2½ hours of video. For thisapplication, a data storage unit is provided to facilitate editing andproduction activities, and it is anticipated that these units would beemployed in much the same way as video cassettes are currently used inBetacam SP and other electronic news gathering (ENG) cameras and invideo productions. This data storage unit may be implemented by use of amagnetic, optical (such as DVD-R or DVD-RAM) discs, or magneto-opticaldisk drive with removable storage media, by a removable disk-drive unit,such as those based on the PCMCIA standards, by tape-based storagemeans, or by semiconductor-based memory. Future advances, in storagetechnology will lead to longer duration program data storage.Alternatively, this storage capacity could be applied to lower ratios ofdata compression, higher sampling precision (10 bits or more) orhigher-pixel-count images, within the limits of the same size media.

FIG. 2 shows the functional diagram for the storage-device-based digitalrecorder employed in the video camera, or separately in editing andproduction facilities. As shown, a removable hard disk drive 70 isinterfaced through a bus controller 72. In practice, alternative methodsof storage such as optical drives (such as DVD-R or DVD-RAM units) ormagneto-optical drives could be used, based on various interface busstandards such as SCSI-2. This disk drive system currently achieves datatransfer rates of 40 MB/sec, and higher rates on these or other datastorage devices, such as high-capacity removable memory modules, isanticipated. If a digital tape-based format is selected, a tape drive 88is interfaced through the bus controller 72. Currently available digitaltape-based formats include DVCPRO, DVCPRO50, DVCAM, Betacam SX, DigitalS50, and others. These units typically offer storage capacities in therange of 30 to 50 GigaBytes. The microprocessor 74 controls the 64-bitor wider data bus 80, which integrates the various components. Currentlyavailable microprocessors include the Alpha 21164 by Digital EquipmentCorporation, or the MIPS processor family by MIPS Technologies, Inc.Future implementations would rely on the Pentium™ series by Intel Corp.or the PowerPC G3, which is capable of sustained data transfer rates of100 MB/sec.

Up to 256 MB of ROM, shown at 76, is anticipated for operation, as is256 MB or more of RAM, shown at 78. Current PC-based video productionsystems are equipped with at least 64 MB of RAM, to allow sophisticatedediting effects. The graphics processor 82 represents dedicated hardwarethat performs the various manipulations required to process the inputvideo signals 84 and the output video signals 86. Although shown usingan RGB format, either the inputs or outputs could be configured inalternative signal formats, such as Y/R-Y/B-Y, YIQ, YUV or othercommonly used alternatives. In particular, while a software-basedimplementation of the processor 82 is possible, a hardware basedimplementation is preferred, with the system employing a compressionratio of 5:1 for the conventional/widescreen signals(“NTSC/PAL/Widescreen”), and a 10:1 compression ratio for HDTV signals(1280×720 or 1920×1080, as described herein above). Example of the manyavailable options for this data compression include the currentlyavailable Motion-JPEG system and the MPEG systems. Image re-sizingalternatively may be performed by dedicated microprocessors, such as thegm865X1 or gm833X3 by Genesis Microchip, Inc. Audio signals may beincluded within the data stream, as proposed in the several systems fordigital television transmission considered by the Federal CommunicationsCommission, or by one of the methods available for integrating audio andvideo signals used in multi-media recording schemes, such as theMicrosoft “AVI” (Audio/Video Interleave) file format. As an alternative,an independent system for recording audio signals may be implemented,either by employing separate digital recording provisions controlled bythe same system and electronics, or by implementing completely separateequipment external to the camera system described herein above.

FIG. 3 shows the components that comprise a multi-format audio/videoproduction system according to the invention. As in the case of thecomputer disk- or tape-based recording system of FIG. 2, an interfacebus controller 106 provides access to a variety of storage devices,preferably including an internal hard-disk drive 100, a tape-drive 102,and a hard-disk drive with removable media or a removable hard-diskdrive 104. Other possible forms of high-capacity data storage (notshown) utilizing optical, magneto-optical, or magnetic storagetechniques may be included, as appropriate for the particularapplication. The interface bus standards implemented could include,among others, SCSI-2. Data is transmitted to and from these devicesunder control of microprocessor 110. Currently, data bus 108 wouldoperate as shown as 64-bits wide, employing microprocessors such asthose suggested for the computer-disk-based video recorder of FIG. 3. Ashigher-powered microprocessors become available, such as the PowerPC G3,the data bus may be widened to accommodate 128 bits, and the use ofmultiple parallel processors may be employed, with the anticipated goalof 1,000 MIPS per processor. Up to 256 MB of ROM 112 is anticipated tosupport the requisite software, and at least 1,024 MB of RAM 114 willallow-for the sophisticated image manipulations, inter-frameinterpolation, and intra-frame interpolation necessary for sophisticatedproduction effects, and for conversions between the various imageformats.

A key aspect of the system is the versatility of the graphics processorshown generally as 116. Eventually, dedicated hardware will allow thebest performance for such operations as image manipulations andre-scaling, but it is not a requirement of the system that it assumethese functions, or even that all of these functions be included in thegraphics processor in every configuration of the system. Three separatesections are employed to process the three classifications of signals.Although the video input and output signals described herein below areshown, by example, as RGB, any alternative format for video signals,such as Y/R-Y/B-Y, YIQ, YUV, or other alternatives may be employed aspart of the preferred embodiment. One possible physical implementationwould be to create a separate circuit board for each of the sections asdescribed below, and manufacture these boards so as to be compatiblewith existing or future PC-based electrical and physical interconnectstandards.

A standard/widescreen video interface 120, intended to operate withinthe 1024×576, 1280×720, 1024×768, 854×480, 640×480 or 1280×960 imagesizes, accepts digital RGB or Y/R-Y/B-Y signals for processing andproduces digital RGB or Y/R-Y/B-Y outputs in these formats, as showngenerally at 122. Conventional internal circuitry comprising D/Aconverters and associated analog amplifiers are employed to convert theinternal images to a second set of outputs, including analog RGB orY/R-Y/B-Y signals and composite video signals. These outputs mayoptionally be supplied to either a conventional multi-scan computervideo monitor or a conventional video monitor having input provisionsfor RGB or Y/R-Y/B-Y signals (not shown). A third set of outputssupplies analog Y/C video signals. The graphics processor may beconfigured to accept or output these signals in the standard NTSC, PAL,or SECAM formats, and may additionally be utilized in other formats asemployed in medical imaging or other specialized applications, or forany desired format for computer graphics applications. Conversion ofthese 24 frame-per-second progressive images to the 30 fps Interlaced(actually, 29.97 fps) NTSC and 25 fps PAL formats may be performed in asimilar manner to that used for scanned film materials, that is, to NTSCby using the conventional 3:2 “pull-down” field-sequence, or to PAL byreproducing the images at the higher 25 fps rate.

If the source signal is 24 fps interlaced, these images first arede-interlaced to 48 fps progressive, which can be performed by dedicatedmicroprocessors such as the gmVLD8 or gmVLD10 by Genesis Microchips, andthen converted to 60 fps progressive by utilizing a “Fourth FrameRepeat” process (which repeats the fourth frame in every sequence).Next, the signal is interlaced to produced 60 fps interlaced, and halfof the fields are discarded to produce 30 fps interlaced (as disclosedin FIG. 7F). If the source format is 25 fps interlaced video (as wouldresult from using conventional PAL-type equipment, or PAL-type equipmentas modified in accordance with the invention), the first step is to slowdown the frame rate by replaying the signal at 24 fps Interlaced. Next,the signal is de-interlaced to 48 fps progressive (as described hereinabove), and the Fourth Frame Repeat process is utilized to convertthe-signal to 60 fps progressive. In the last step, the signal isinterlaced to produced 60 fps interlaced, and half of the fields arediscarded to produce 30 fps interlaced. Alternatively, if the sourcesignal is 24 fps progressive, the 60 fps progressive signal may beproduced directly from a “3:2 Frame Repeat” process shown in FIG. 7G(which is analogous to the conventional “3:2 pull-down” field-sequencingprocess previously described). For other HDTV frame rates, aspectratios, and line rates, intra-frame and inter-frame interpolation andimage conversions may be performed by employing comparable techniqueswell known in the art of computer graphics and television.

An HDTV video interface 124, intended to operate within the 1920×1080 orother larger image sizes (with re-sizing as necessary), accepts digitalRGB or Y/R-Y/B-Y (or alternative) signals for processing and produces,digital outputs in the same image format, as shown generally at 126. Asis the case for the standard/widescreen interface 120, conventionalinternal circuitry comprising D/A converters and associated analogamplifiers are employed to convert the internal images to a second setof outputs, for analog RGB signals and composite video signals. Inalternative embodiments, this function may be performed by an externalupconvertor, which will process the wideband signal of the instantinvention. A modification of currently available upconvertors isrequired, to increase the frequency of the sampling clock in order topreserve the full bandwidth of this signal, in accordance with theinvention. In this case, frequency of the sampling clock is preferablyadjustable to utilize one of several available frequencies.

The third section of the graphics processor 116 shown in FIG. 3 is thefilm output video interface 128, which comprises a special set of videooutputs 130 intended for use with devices such as laser film recorders.These outputs are preferably configured to provide a 3840×2160 or otherlarger image size from the image sizes employed internally, usingre-sizing techniques discussed herein as necessary for the formatconversions. Although 24 fps is the standard frame rate for film, someproductions employ 30 fps (especially when used with NTSC materials) or25 fps (especially when used with PAL materials), and these alternativeframe rates, as well as alternative image sizes and aspect ratios forinternal and output formats, are anticipated as suitable applications ofthe invention, with “3:2-pull-down” utilized to convert the internal 24fps program materials to 30 fps, and 25 fps occurring automatically asthe film projector runs the 24 fps films at the 25 fps rate utilized forPAL-type materials.

Several additional optional features of this system are disclosed inFIG. 3. The graphics processor preferably also includes a special output132 for use with a color printer. In order to produce the highestquality prints from the screen display it is necessary to adjust theprint resolution to match the image resolution, and this isautomatically optimized by the graphics processor for the various imagesizes produced by the system. In addition, provisions may be includedfor an image scanner 134, which may be implemented as a still imagescanner or a film scanner, thereby enabling optical images to beintegrated into the system. An optional audio processor 136 includesprovisions for accepting audio signals in either analog or digital form,and outputting signals in either analog or digital form, as shown in thearea generally designated as 138. For materials including audiointermixed with the video signals as described herein above, thesesignals are routed to the audio processor for editing effects and toprovide an interface to other equipment.

It is important to note that although FIG. 3 shows only one set of eachtype of signal inputs, the system is capable of handling signalssimultaneously from a plurality of sources and in a variety of formats.Depending on the performance level desired and the image sizes and framerates of the signals, the system may be implemented with multiple harddisk or other mass-storage units and bus controllers, and multiplegraphics processors, thereby allowing integration of any combination oflive camera signals, prerecorded materials, and scanned images. Improveddata compression schemes and advances in hardware speed will allowprogressively higher frame rates and image sizes to be manipulated inreal-time.

Simple playback of signals to produce PAL output is not a seriousproblem, since any stored video images may be replayed at any frame ratedesired, and filmed material displayed at 25 fps is not objectionable.Indeed, this is the standard method for performing film-to-tapetransfers used in PAL- and SECAM-television countries. Simultaneousoutput of both NTSC and film-rate images may be performed by exploitingthe 3:2 field-interleaving approach: 5×24=120=2×60. That is, two filmframes are spread over five video fields. This makes it possible toconcurrently produce film images at 24 fps and video images at 30 fps.The difference between 30 fps and the exact 29.97 fps rate of NTSC maybe palliated by slightly modifying the system frame rate to 23.976 fps.This is not noticeable in normal film projection, and is an acceptabledeviation from the normal film rate.

The management of 25 fps (PAL-type) output signals in a signaldistribution system configured for 24 fps production applications (orvice versa) presents technical issues which must be addressed, however.One alternative for facilitating these and other frame-rate conversionsis explained with reference to FIG. 4. A digital program signal 404 isprovided to a signal compression circuit 408. If the input programsignal is provided in analog form 402, then it is first processed by A/Dconverter 406 to be placed in digital form. The signal compressor 408processes the input program signal so as to reduce the effective datarate, utilizing any of the commonly implemented data compressionschemes, such as motion-JPEG, MPEG1, MPEG2, etc. well known in the art.As an alternative, the digital program signal 404 may be provided indata-compressed form. At this point, the digital program signal isprovided to data bus 410. By way of example, several high-capacitydigital storage units, designated as “storage means A” 412 and “storagemeans B” 414, are included for storing the digital program signalspresented on data bus 410, under management by controller 418.

The two storage means 412 and 414 may be used in alternating fashion,with one storing the source signal until it reaches its full capacity.At this point, the other storage means would continue storing theprogram signal until it, too, reached its full capacity. The maximumprogram storage capacity for the program signals will be determined byvarious factors, such as the input program signal frame rate, the framedimensions in pixels, the data compression rate, the total number andcapacities of the various storage means, and so forth. When theavailable storage capacity has been filled, this data storage schemeautomatically will result in previously-recorded signals beingoverwritten. As additional storage means are added, the capacity fortime-delay and frame rate conversion is increased, and there is norequirement that all storage means be of the same type, or of the samecapacity. In practice, the storage means would be implemented using anyof the commonly available storage techniques, including, for example,magnetic disks, optical (such as DVD-RAM discs) or magneto-opticaldiscs, or semiconductor memory.

When it is desired to begin playback of the program signal, signalprocessor 416, under management by controller 418 and through userinterface 420, retrieves the stored program signals from the variousstorage means provided, and performs any signal conversions required.For example, if the input program signals were provided at a 25 fps rate(corresponding to a 625-line broadcast system), the signal processorwould perform image resizing and inter-frame interpolation to convertthe-signal to 30 fps (corresponding to a 525-line broadcast system).Other conversions (such as color encoding system conversion fromPAL-format to NTSC, etc., or frame dimension or aspect-ratio conversion)will be performed as necessary. The output of the signal processor isthen available in digital form as 422, or may be processed further, intoanalog form 426 by D/A converter 424. In practice, a separate data bus(not shown) may be provided for output signals, and/or the storage meansmay be implemented by way of dual-access technology, such as dual-portRAM utilized for video-display applications, or multiple-head-accessdisk or disk storage units, which may be configured to providesimultaneous random-access read and write capabilities. Wheresingle-head storage means are implemented, suitable input buffer andoutput buffer provisions are included, to allow time for physicalrepositioning of the record/play head.

In utilizing program storage means including synchronous recording andplayback capabilities of the types just described, if it is known that aprogram will be stored in its entirety before the commencement ofplayback, that is, with no time-overlap existing between the occurrenceof the input and output signal streams, it typically will be mostefficient to perform any desired frame conversion on the program eitherbefore or after initial storage, depending upon which stored formatwould result in the least amount of required memory. For example, if theprogram is input at a rate of 24 frames per second, it probably will bemost efficient to receive such a program and store it at that rate, andperform a conversion to higher frame rates upon output. In addition, insituations where a program is recorded in its entirety prior toconversion into a particular output format, it is most efficient tostore the program either on a tape-based format or a format such as thenew high-capacity DVD-type discs, given the reduced cost, on a per-bitbasis, of these types of storage. Of course, conventional high-capacitydisk storage also may be used, and may become more practical as storagecapacities continue to increase and costs decrease. If it is known thata program is to be output at a different frame rate while it is beinginput or stored, it is most preferable to use disk storage and toperform the frame rate conversion on an ongoing basis, using one of thetechniques described above. In this case, the high-capacity videostorage means, in effect, assumes the role of a large video bufferproviding the fastest practical access time. Again, other memory means(types) may be used, including all solid-state and semiconductor types,depending upon economic considerations, and so forth.

As an example of an alternative embodiment, the storage means 100 or 104are equipped with dual-head playback facilities and a second set ofgraphics processing hardware (not shown) analogous in function to thenormal graphics processing hardware (identical to the standard hardwareshown as 120, 124, and 128), and having analogous signal outputfacilities (identical to the standard provisions shown as 122, 126, 130,and 132). In this case, the two heads would be driven independently, toprovide simultaneous, asynchronous playback at different frame rates.That is, one head would be manipulated so as to provide a data streamcorresponding to a first frame rate (for example, 25 fps), while thesecond head would be manipulated so as to provide a data streamcorresponding to a second frame rate (for example, 24 fps, which, inturn, may be converted to 30 fps, using the “3:2-pull-down” technique).In this case, both the storage means and also the internal bus structureof the system would have to support the significantly increased datarate for providing both signal streams simultaneously, or, as analternative, a second, separate data bus would be provided.

In some applications, a more sophisticated conversion scheme isrequired. For example, in frame rate conversion systems of conventionaldesign, if an input program signal having a 24 fps rate format is to bedisplayed at a 25 fps rate, it is customary to simply speed up thesource signal playback, so as to provide the signals at a 25 fps rate.This is the procedure utilized for performing a conversion of24-fps-film-material for 25 fps PAL-format video usage. However,implementation of this method requires that the user of the outputsignal must have control over the source-signal playback. In a wide-areadistribution system (such as direct-broadcast-satellite distribution)this is not possible. While a source signal distributed at 24 fpsreadily could be converted to 30 fps (utilizing the familiar“3-2-pull-down” technique), the conversion to 25 fps is not as easilyperformed, due to the complexity and expense of processing circuitryrequired for inter-frame interpolation over a 24-frame sequence.However, utilizing the system disclosed in FIG. 4, the conversion isstraightforward. If, for example, a 24 fps program lasting 120 minutesis transmitted in this format, there are a total of 172,800 frames ofinformation (24 frames/second×60 seconds/minute×120 minutes). Display ofthis program in speeded-up fashion at 25 fps would mean that the inputframe rate falls behind the output frame rate by one frame per second,or a total of 7,200 frames during the course of the program. At a 24 fpstransmission rate, this corresponds to 300 seconds transmission time. Inother words, for the input program (at 24 fps) and the output program(at 25 fps) to end together, the input process would have to commence300 seconds before the output process begins. In order to perform thisprocess, then, it is necessary for the storage means to have thecapacity to retain 300 seconds of program material, in effect serving asa signal buffer. As an example, for the systems disclosed herein inwhich the compressed-data rates range from 5.5 MB/sec (for 24 fpsstandard/widescreen Y/R-Y/B-Y-based TV formats, using 5:1 datacompression such as MPEG or motion-MEG and 4:2:2 processing with 8-bitprecision) to 10 MB/sec (for 24 fps HDTV Y/R-Y/B-Y-based formats, using10:1 data compression such as MPEG or motion-MEG and 4:2:2 processingwith 8-bit precision), it may be necessary to store as much as 3.3GBytes of data, which is readily available by way of multiple disks ordiscs utilizing conventional storage technology. In practice, thetransmission simply would begin 300 seconds before the playback begins,and once the playback starts, the amount of buffered signal woulddecrease by one frame per second of playback until the last signal ispassed through as soon as it is received.

A mirror of this situation arises in the case of a 25 fps signal to bedisplayed at 24 fps, or some other data rate readily provided byconversion from 24 fps (such as 30 fps). In this case, the source signalis provided at a higher frame rate than the output signal, so that aviewer watching a program from the onset of the transmission would fallbehind the source signal rate, and the storage means would be requiredto hold frames of the program to be displayed at a time after the sourcesignal arrival time. In the case of the 120 minute program describedabove, the viewing of the source program would conclude 300 secondsafter the source signal itself had concluded, and comparablecalculations are applied for the storage means. In this case, the extraframes would be accumulated as the buffer contents increased, until,after the transmission has completed, the last 300 seconds would bereplayed directly from the storage means.

The conversion of frame rates from 30 fps to 24 fps or to 25 fps is morecomplicated, because some form of inter-frame interpolation is required.In one case, a multi-frame storage facility would allow this type ofinterpolation to be performed in a relatively conventional manner, astypically is utilized in NTSC-to-PAL conversions (30 fps to 25 fps). Atthis point, a 25 fps to 24 fps conversion could he performed, inaccordance with the methods and apparatus described herein above.

It should be noted that if, for example, a DVD-R-type, DVD-RAM-type, orsome form of removable magnetic storage media is selected, then theimplementation of the significantly higher data compression rates ofMPEG-2 coding techniques will result in the ability to record an entireprogram of 120 minutes or more in duration. In this manner, the completeprogram is held in the disk/buffer, thereby enabling the user to performtrue time-shifting of the program, or allowing the program rights ownerto accomplish one form of software distribution, in accordance with theinvention.

An alternative method to carry out this frame rate conversion is carriedout utilizing the following process. The 30 fps interlaced signal isfirst de-interlaced to 60 fps Progressive. Then, every fifth frame isdeleted from the sequence, producing a 48 fps progressive signal stream.Next, these remaining frames are converted to 24 fps interlaced, asdisclosed in FIG. 7I (“5^(th) Frame Reduction”). If the original sourcematerial were from 24 fps (for example, film), then if the repeatedfields (i.e., the “3” field of the 3:2 sequence) were identified at thetime of conversion, then the removal of these fields would simply returnthe material to its original form. If the desired conversion is to befrom 30 fps to 25 fps, then an equivalent procedure would be performedusing the storage-based frame-conversion method described herein above.As an alternative, the 30 fps interlaced signal would first bede-interlaced to 60 fps progressive; then, every sixth frame would bedeleted from the sequence (“6^(th) Frame Reduction”). The remainingframes are re-interlaced to produce 25 fps interlaced, as disclosed inFIG. 7H. Depending on the original source material frame rate andintermediate conversions, the user would select the method likely topresent the least amount of image impairment.

In the case in which the user is able to exercise control over the framerate of the source program material, an alternative method is available.Just as film-to-video transfers for PAL-format (25 fps) presentationsutilize a speeded-up playback of the 24 fps film materials to sourcethem at the 25 fps Progressive rate (thereby matching the intendedoutput frame rate), the reverse of this process enables a user toutilize materials originated at 25 fps Progressive to produce playbackat 24 fps. As disclosed herein above, conversions of 24 fps progressivematerials are handled easily by way of conventional methods (such as the“3:2-pull-down” method), and therefore the operator control of thesource material enables the user to utilize materials originating fromconventional or widescreen PAL format sources for editing andproduction, then replay the resulting program at 24 fps for conversionto either standard or widescreen NTSC output materials, or even to HDTVformat materials, all at 30 fps Interlaced, by performing the“3:2-pull-down” process.

If the source format is 25 fps interlaced video (as would result fromusing conventional PAL-type CCD widescreen camera), an alternativemethod for producing a 30 fps Interlaced signal is available. Instead ofperforming a slow-down to produce a 24 fps interlaced signal, the 25 fpsInterlaced signal is first de-interlaced to 50 fps progressive. Next, a“4^(th) Frame Repeat” process is applied, which results in a 62.5 fpsprogressive signal. This signal is then converted to 62.5 fpsinterlaced, and after half of the fields are discarded, to produce 31.25fps interlaced. After data compression, the signal undergoes a slow-downprocess, resulting in a 30 fps interlaced signal which now has acompressed-data-rate of less than 10 Mbytes per second, as disclosed inFIG. 7D. By using this procedure, the entire process from the CCD camerato the final conversion to 30 fps Interlaced only one data compressionstep is employed. Alternatively, if the output of the camera is alreadyin data compressed form, then this signal must be decompressed beforeapplying the listed conversion steps. In order to ensure accurateconversion, interlace and de-interlace processes should only be appliedto de-compressed signals. Conversely, speed-up and slow-down proceduresare preferably applied with compressed data, as the raw data rate foruncompressed video, depending on the image dimensions in pixels andframe rate, will be in the range of 30 to 100 MB per second, which isnot practical for current technology storage devices.

A variety of conversions between formats (both interlaced andprogressive) having differing frame rates, and some of these possibleconversion paths are indicated in FIGS. 7A through 7I. While extensive,these listings are not intended to represent a complete listing of allalternatives, as in many cases there is more than one combination ofmethods which may affect an equivalent conversion. Depending on theparticular application, different paths may be selected, and thesediffering paths may produce more, or less, effective results.

The various alternatives utilize several techniques not previouslyapplied to these types of conversions. For example, conversions of 60fps progressive signals to 30 fps Progressive may be effected by simplydropping alternate frames. On the other hand, a “3:2 Frame Repetition”method consists of repeating a first frame a second and a third time,then repeating the next frame a second time, thereby converting twoframes into five frames (as depicted in FIG. 7G).

Depending on whether the source material is 24 fps progressive or 24 fpsinterlaced, different approaches are utilized for conversion to 30 fpsinterlaced. In the first case, the 24 fps progressive signal is firstconverted to 24 fps Interlaced. A set of four consecutive frames may beindicated as 1A1B, 2A2B, 3A3B, 4A4B. By recombining these fields (butoutputting them at a 30 fps rate) the following field sequence isobtained: 1A1B, 1A2B, 2A3B, 3A4B, 4A4B. This sequence repeats for everyfour input frames, which is to say, for every five output frames (asdepicted in FIG. 7C).

Alternatively, for a signal which originates at 24 fps Interlaced, theoriginal four-frame sequence is identical. However, the situation ismore complicated because the absolute time-sequence of frames must bepreserved. For this reason, it is necessary to reverse the fieldidentification of alternate groups of fields in order to preserve theproper interlace relationship between the fields. In effect, everyfourth and seventh field in the eight-field (24 fps interlaced) sequenceis repeated, but with reversed field identification (as disclosed inFIG. 7E). When the fourth input field has had its identificationreversed (to produce the fifth output field), then the next two inputfields (corresponding to the sixth and seventh output field) in thesequence also will require field reversal, in order to preserve thecorrect sequence for proper interlace. Furthermore, when the seventhinput field is repeated, the first time it will appear inreversed-field-identity from as the eighth output field. For thisprocedure, the resulting field sequence will be 1A1B, 2A2B, 2B*3A*,3B*4A*, 4A4B (wherein a field having reversed field identification isdenoted by a * symbol). This sequence repeats for every four inputframes, which is to say, for every five output frames.

In addition, the reversal of the field identity of the fourth inputfield (when repeated) results in information that previously wasdisplayed on the second scan line now being displayed on the first scanline. Therefore, it is necessary to discard the first line of the nextreversed-field, so that the information displayed on the second scanline of the new field will be the information previously displayed onthe third line of the next (reversed) field. After the seventh inputfield has been reversed (to produce the eighth output field, thefollowing fields are once again in the proper line order without anyfurther adjustments of this kind (as disclosed in FIG. 7E).

For image manipulations entirely within the internal storage format,there is no issue as to interlacing, as the graphics processor is onlymanipulating a rectangular array of image pixels, not individual scanlines. As such, identification of fields is derived solely from thelocation of the image pixels on either odd-numbered lines oreven-numbered lines. The interlacing field identification adjustmentsare made only at the time of output to the display device. In theseapplications, the presence of the storage means allows the viewer tocontrol the presentation of a program, utilizing a user interface 420 tocontrol the playback delay and other characteristics of the signal whileit is being stored or thereafter. In practice, a wide range ofalternatives for input frame rates and output frame rate conversions aremade available through this system, by selecting the most appropriate ofthe various methods for altering the frame rate of a signal describedherein.

FIG. 5 shows the inter-relationship of the various film and videoformats compatible with the invention, though not intended to beinclusive of all possible implementations. In typical operations, themulti-format audio/video production system 162 would receive film-basedelements 160 and combine them with locally produced materials already inthe preferred internal format of 24 frames-per-second. In practice,materials may be converted from any other format including video at anyframe rate or standard. After the production effects have beenperformed, the output signals may be configured for any use required,including, but not limited to, HDTV at 30/60 fps shown as 164,widescreen at 30 fps shown as 166, widescreen at 25 fps shown as 170, orHDTV at 25/50 fps shown as 172. In addition, output signals at 24 fpsare available for use in a film-recording unit 168.

In FIG. 6, signals are provided from any of several sources, includingconventional broadcast signals 210, satellite receivers 212, andinterfaces to a high bandwidth data network 214. These signals would beprovided to the digital tuner 218 and an appropriate adapter unit 220for access to a high-speed data network before being supplied to thedecompression processor 222. As an option, additional provisions fordata compression would provide for transmission of signals from thelocal system to the high bandwidth data network 214. The processor 222provides any necessary data de-compression and signal conditioning forthe various signal sources, and preferably is implemented as a plug-incircuit board for a general-purpose computer, though the digital tuner218 and the adapter 220 optionally may be included as part of theexisting hardware.

The output of processor 222 is provided to the internal data bus 226.The system microprocessor 228 controls the data bus, and is providedwith 32 to 128 MB of RAM 230 and up to 64 Mb of ROM 232. Thismicroprocessor could be implemented using one of the units previouslydescribed, such as the PowerPC 604, PowerPC G3, Pentium-series, or otherprocessors. A hard disk drive controller 234 provides access to variousstorage means, including, for example, an internal hard disk drive unit236, a removable hard disk drive unit 238, a unit utilizing removablemagnetic, optical, or magneto-optical media (not shown), or a tape drive240. These storage units also enable the PC to function as a videorecorder, as described above. A graphic processor 242, comprisingdedicated hardware which optionally be implemented as a separate plug-incircuit board, performs the image manipulations required to convertbetween the various frame sizes (in pixels), aspect ratios, and framerates. This graphics processor uses 16 to 32 MB of DRAM, and 2 to 8 MBof VRAM (depending on the type of display output desired. For frame sizeof 1280×720 with an aspect ratio 16:9, the lower range of DRAM and VRAMwill be sufficient, but for a frame size of 1920×1080, the higher rangeof DRAM and VRAM is required. In general, the 1280×720 size issufficient for conventional “multi-sync” computer display screens up to20 inches, and the 1920×1080 size is appropriate for conventional“multi-sync” computer display screens up to 35 inches. Analog videooutputs 244 are available for these various display units. Using thissystem, various formats may be displayed, including (for 25 fps, shownby speeding up 24 fps signals) 768×576 PAL/SECAM, 1024×576 wide-screen,and 1280×720/1920×1080 HDTV, and (for 30 and 60 fps, shown by utilizingthe well-known “3:2 pull-down” technique, and for 29.97 fps, shown by aslight slow-down in 30 fps signals) 640×480 NTSC and 854×480wide-screen, and 1920×1080 NHK (Japan) HDTV.

It will be appreciated by the skilled practitioner that most of thehighest quality program material has been originated on 24 fps 35-mmfilm, and therefore conversions that rely on reconstituting the signalmaterial from 25 fps or 30 fps materials into 24 fps material do notentail any loss of data or program material. In addition, signals thathave been interlaced from a lower or equivalent frame rate source signalin any of the currently available means (24 fps to 25 fps via speed-up;24 fps to 30 fps via “3:2-pull-down”) may be de-interlaced andreconstituted as progressive-scan frames without introducing any signalartifacts, provided that the original frames are recreated from properlymatched fields. If it is desired to produce 24 fps interlaced, 25 fpsInterlaced, or 30 fps interlaced signals from higher frame rateprogressive signals (such as 48 fps Progressive, 50 fps progressive, or60 fps progressive signals, respectively) these may be obtained byinterlacing these signals and discarding the redundant data.Alternatively, if it is desired to produce 24 fps progressive, 25 fpsprogressive, 30 fps Progressive, or 48 fps progressive signals fromhigher frame rate progressive signals (such as 48 fps progressive, 50fps progressive, 60 fps progressive, or 96 fps progressive signals,respectively), these may be obtained by applying a 2:1 frame reduction.These techniques are summarized in FIG. 7A, with conversion chartsshowing typical process flow charts in FIGS. 7B and 7C.

FIG. 8 shows one possible implementation of a universal playback device,in accordance with the invention. By way of example, a DVD-type videodisk 802 is rotatably driven by motor 804 under control of speed-controlunit 806. One or more laser read- or read/write-heads 808 are positionedby position control unit 810. Both the speed control unit and theposition control unit are directed by the overall system controller 812,at the direction of the user interface 814. It should be noted that thenumber and configuration of read- or read/write-heads will be determinedby the choice of the techniques employed in the various embodimentsdisclosed herein above. The signals recovered from the laser heads isdelivered to signal processor unit 820, and the data stream is splitinto an audio data stream (supplied to audio processor unit 822) and avideo data stream (supplied to video graphics processor unit 830).During the audio recovery process, the alteration of the playback framerate (for example, from 24 fps to 25 fps, accomplished by speed controladjustment) may suggest the need for pitch-correction of the audiomaterial. This procedure, if desired, may be implemented either as partof the audio processor 822, or within a separate, external unit (notshown), as offered by a number of suppliers, such as Lexicon.

The video data stream may undergo a number of modifications within thegraphics processor, shown generally at 830, depending on the desiredfinal output format. Assuming that the output desired is NTSC or someother form of wide-screen or HDTV signal output at a nominal frame rateof 30 fps, a signal sourced from the disk at 24 fps would undergo a“3:2-pull-down” modification as part of the conversion process (asexplained herein above). If the signal as sourced from the disk is basedon 25 fps, then it would undergo an preliminary slowdown to 24 fpsbefore the “3:2-pull-down” processing is applied. It should be notedthat the 0.1% difference between 30 fps and 29.97 fps only requires thebuffering of 173 frames of video over the course of a 120-minuteprogram, and at a data rate of 5.5 MB/sec, this corresponds toapproximately 39 MB of storage (for standard/widescreen) or 79 MB ofstorage (for HDTV), which readily may be implemented insemiconductor-based memory. In any event, a signal supplied to thegraphics processor at a nominal 24 fps simultaneously may be output atboth 30 fps and 29.97 fps, in image frames compatible with both NTSC andNTSC/widescreen (the standard/widescreen video interface 832), and HDTV(HDTV video interface 834), in accordance with the invention asdescribed herein above.

As disclosed above, an optional film output video interface 836 may beincluded, with digital video outputs for a film recorder. Overall, theoutputs for the graphics processor 830 parallel those of theMulti-Format Audio/Video Production System as shown in FIG. 5 anddisclosed herein above. In addition, for signals to be output in aformat having a different aspect ratio than that of the source signal,it may be necessary to perform a horizontal and/or vertical “pan/scan”function in order to assure that the center of action in the sourceprogram material is presented within the scope of the output frame. Thisfunction may be implemented within the graphics processor by utilizing a“tracking” signal associated with the source program material, forexample, as part of the data stream for each frame, or, alternatively,through a listing identifying changes that should be applied during thepresentation of the source material. Where no “tracking” information isavailable, the image frame would be trimmed along the top and bottom, orthe sides, as necessary in order to fit the aspect ratio of the sourcematerial to the aspect ratio of the output frame. This latter techniqueis explained herein above, with reference to FIGS. 1A-1D. In addition,the program material may include security information, such as regionalor geographical information directed towards controlling the viewing ofthe program material within certain marketing areas or identifiableclasses of equipment (such as hardware sold only in the United States orin the German market). This information, as has been disclosed for usewith other disk-n and tape-based systems, often relates to issues suchas legal licensing agreements for software materials. It may beprocessed in a way similar to the detection and application of the“pan/scan” tracking signal, and the signal processor 820, under thedirection of controller 812 may act to enforce these restrictions.

Alternatively, if output at 25 fps is desired, it is a simple matter toconfigure the various components of this system to replay the videoinformation of the disk 802 at this higher frame rate. The controllerwill configure the speed control unit 806 (if necessary) to drive themotor 804 at a greater rotational speed to sustain the increased datarate associated with the higher frame rate. The audio processor 822, ifso equipped, will be configured to correct for the change in pitchassociated with the higher frame rate, and the graphics processor willbe configured to provide all output signals at the 25 fps frame rate. AsAlternate method for audio pitch correction, additional audio data canbe stored in disk which is already corrected. When the frame rate ischanged, the corresponding audio data is selected in accordance with theinvention.

As yet another alternative, materials produced at 25 fps and stored onthe disk-based mass storage means of this example could originate fromconventional standard or widescreen PAL format signals. Utilizing theslow-down method, these signals are readily converted to 24 fps framerate, from which conversion to various 30 fps formats is implemented, asdisclosed herein above. This feature has significance in the commercialdevelopment of HDTV, as the ability to utilize more-or-less conventionalPAL format equipment greatly facilitates the economical production andorigination of materials intended for HDTV markets.

A wide range of output frame rates may be made available throughcombination of the techniques of speed-up, slow-down, “3-2-pull-down,”and other related field-rearrangement, de-interlacing,interlacing/de-interlacing, frame repetition, and frame reductiontechniques, as disclosed herein above with respect to FIG. 4 and FIGS.7A-7E, and these various combinations and approaches should beconsidered to be within the scope of the invention. In addition, thesetechniques may be combined with hardware and/or software which performimage manipulations such as line-doubling, line-quadrupling,deinterlacing, etc., such that the display device will be capable ofproviding smoother apparent motion, by increasing the display ratewithout increasing the actual data/information rate. One example wouldbe to process the 24 fps signal from the internal format to convert itinto a 48 fps signal, using field-doubling techniques such asdeinterlacing and line doubling. Then, the process would employframe-store techniques to provide a frame-repeated output at a rate of96 fps. These types of display-related improvements, in conjunction withthe instant invention, should also be considered to be within the scopeof the invention as disclosed herein. Examples of these variouscombinations and conversion methods are included in the table of FIG. 7Aand the chart of FIG. 7E.

In general, the features as described need not all be provided in asingle unit, but rather may be distributed through various externalunits (such as external data-recorders or display units). In addition,particular configurations of the system may include only the graphicscapabilities required for that application (such as the use of 25 fpsPAL outputs, but not 30 fps NTSC) and may even exclude certain options(such as printer outputs), and these variations should be considered tobe within the scope of the invention.

1. A method of producing an audio/video recording, comprising the stepsof: receiving an input video image at a portable audio/video recordingdevice; converting the input video image into a digital productionformat by sampling the input program at a sampling frequency in excessof 18 megahertz; providing a high-capacity digital video storageequipped with an asynchronous program recording and reproducingcapability performing a frame-rate conversion; and processing therecorded video program in the production format using the high-capacityvideo storage on a selective basis outputting a version of the videoprogram having a desired frame rate and image dimensions in pixels. 2.The method of claim 1, wherein the portable audio/video recording deviceis a camcorder.
 3. The method of claim 1, wherein the portableaudio/video recording device is a portable communication device.
 4. Aportable audio/video recording device, comprising: an image sensor,having pixel dimensions of at least 1024×576; a graphics processorconverting the input video image into a digital production format bysampling the input program at a sampling frequency in excess of 18megahertz; a high-capacity digital video storage equipped with anasynchronous program recording and reproducing capability and performinga frame-rate conversion; and output circuitry processing the recordedvideo program in the production format using the high-capacity videostorage on a selective basis to output a version of the video programhaving a desired frame rate and image dimensions in pixels.
 5. Thedevice of claim 4, wherein the portable audio/video recording device isa camcorder.
 6. The device of claim 4, wherein the portable audio/videorecording device is a portable communication device.